Uczenie Maszynowe - LAB2b - LIMEĀ¶

Biblioteka LIME: https://github.com/marcotcr/lime (Dokumentacja API: https://lime-ml.readthedocs.io/en/latest/)

Wprowadzenie - pakietyĀ¶

Niezbędne pakiety i moduły na potrzeby wprowadzenia

InĀ [5]:
import json
from functools import partial

import matplotlib.pyplot as plt
import numpy as np
import torch
from lime import lime_image
from PIL import Image
from skimage.segmentation import mark_boundaries
from torchvision import models, transforms

Wprowadzenie - funkcje pomocniczeĀ¶

Funkcja do wczytywania wskazanego obrazka oraz konwersji do palety RGB.

InĀ [6]:
def get_image(path):
    with open(path, 'rb') as f:
        with Image.open(f) as img:
            return img.convert('RGB')

Funkcja do przekształcania obrazka (zwrĆ³conego przez funkcję get_image) w tensor, akceptowalny na wejściu sieci neronowej.

InĀ [7]:
def image_to_tensor(img):    
    transformer = transforms.Compose([
        transforms.Resize((256, 256)),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(
            mean=[0.485, 0.456, 0.406],
            std=[0.229, 0.224, 0.225])
    ])  
    return transformer(img).unsqueeze(0)

Funkcja używana przez LIME, przyjmuje na wejściu zbiĆ³r obrazkĆ³w, a zwraca prawdopodobieństwa klas. Należy ją przekazać do lime_image.LimeImageExplainer().explain_instance przy użyciu partial, jako partial(predict_batch, <model>), gdzie modelem w naszym wypadku będą sieci neuronowe. Przykłady użycia są zawarte w tym notebooku.

InĀ [8]:
def predict_batch(model, images):
    model.eval()
    transformer = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(
            mean=[0.485, 0.456, 0.406],
            std=[0.229, 0.224, 0.225]) 
    ])   
    
    model.eval()
    batch = torch.stack(tuple(transformer(i) for i in images), dim=0)

    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model.to(device)
    batch = batch.to(device)
    
    logits = model(batch)
    probas = torch.nn.functional.softmax(logits, dim=1)
    return probas.detach().cpu().numpy()

Funkcja, ktĆ³ra przekształca obrazek w format akceptowany na wejściu przez LIME. Przykłady użycia są zawarte w tym notebooku.

InĀ [9]:
def lime_transformer(image):
    transformer = transforms.Compose([
        transforms.Resize((256, 256)),
        transforms.CenterCrop(224)
    ])
    return np.array(transformer(image))

LIME jest gÅ‚Ć³wnie wykorzystywane do wyjaśniania predykcji tzw. czarnych skrzynek, czyli modeli nieinterpretowalnych. Idealnymi kandydatami są Głębokie Sieci Neuronowe, dlatego sprĆ³bujemy wyjaśnić niektĆ³re predykcje gotowych modeli.

Model Inception-v3 - przygotowanie danychĀ¶

https://arxiv.org/abs/1512.00567

Plik ./data/imagenet_class_index.json zawiera przypisanie klas obrazkĆ³w do indeksĆ³w. Jest to istotne, ponieważ zwracane wyniki (np. wartości funkcji logit na wyjściu sieci neuronowych) wykorzystują to, zwracając wyniki w zadanej kolejności.

InĀ [10]:
with open("./data/imagenet_class_index.json") as f:
    content = json.load(f)
    index_to_label = {
        int(index): data[1]
        for index, data in content.items()
    }
InĀ [11]:
image_to_classify = get_image("./data/dogs.png")
plt.imshow(image_to_classify)
Out[11]:
<matplotlib.image.AxesImage at 0x2428a17dcd0>
No description has been provided for this image
InĀ [12]:
img_tensor = image_to_tensor(image_to_classify)

Załadowanie pretrenowanego modeluĀ¶

InĀ [13]:
inception_v3 = models.inception_v3(pretrained=True)

c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=Inception_V3_Weights.IMAGENET1K_V1`. You can also use `weights=Inception_V3_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Downloading: "https://download.pytorch.org/models/inception_v3_google-0cc3c7bd.pth" to C:\Users\Monika/.cache\torch\hub\checkpoints\inception_v3_google-0cc3c7bd.pth
100%|ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ| 104M/104M [00:02<00:00, 54.2MB/s]

PredykcjaĀ¶

InĀ [14]:
inception_v3.eval()
logits = inception_v3(img_tensor)

ZwrĆ³Ä‡ uwagę, że model zwraca wartości funkcji logit, a nie prawdopodobieństwa klas, dlatego wyniki trzeba przetworzyć (np. przy użyciu funkcji softmax).

InĀ [17]:
probas = torch.nn.functional.softmax(logits, dim=1)

SprawdÅŗmy N najbardziej prawdopodobnych klas

InĀ [18]:
TOP_N_LABELS = 15

probas_top = probas.topk(TOP_N_LABELS)
top_probas = probas_top[0][0].detach().numpy()
top_labels = probas_top[1][0].detach().numpy()
for proba, label in zip(top_probas, top_labels):
    print(f"Class: {index_to_label[label]:<30} | Probability: {proba:.6f}")

Class: Bernese_mountain_dog           | Probability: 0.935930
Class: EntleBucher                    | Probability: 0.038448
Class: Appenzeller                    | Probability: 0.023756
Class: Greater_Swiss_Mountain_dog     | Probability: 0.001818
Class: Gordon_setter                  | Probability: 0.000009
Class: Blenheim_spaniel               | Probability: 0.000007
Class: English_springer               | Probability: 0.000002
Class: tabby                          | Probability: 0.000002
Class: robin                          | Probability: 0.000001
Class: guinea_pig                     | Probability: 0.000001
Class: amphibian                      | Probability: 0.000001
Class: Japanese_spaniel               | Probability: 0.000001
Class: African_grey                   | Probability: 0.000001
Class: Brittany_spaniel               | Probability: 0.000001
Class: toucan                         | Probability: 0.000001

Teraz możemy te funkcje zebrać razemĀ¶

InĀ [19]:
def get_prediction_probabilities(image, model):
    img_tensor = image_to_tensor(image)
    model.eval()
    logits = model(img_tensor)
    probas = torch.nn.functional.softmax(logits, dim=1)
    
    TOP_N_LABELS = 15

    probas_top = probas.topk(TOP_N_LABELS)
    top_probas = probas_top[0][0].detach().numpy()
    top_labels = probas_top[1][0].detach().numpy()
    for proba, label in zip(top_probas, top_labels):
        print(f"Class: {index_to_label[label]:<30} | Probability: {proba:.6f}")

I sprawdzić jak ta predykcja wygląda dla innego obrazkaĀ¶

InĀ [20]:
exercise_image = get_image("./data/cat_mouse.jpeg")
plt.imshow(exercise_image)
Out[20]:
<matplotlib.image.AxesImage at 0x2428a6ca510>
No description has been provided for this image

Zadanie: sprawdÅŗ jak będzie wyglądała predykcja dla powyższego obrazkaĀ¶

InĀ [21]:
get_prediction_probabilities(exercise_image, inception_v3)

Class: Egyptian_cat                   | Probability: 0.967492
Class: tabby                          | Probability: 0.024167
Class: lynx                           | Probability: 0.005490
Class: tiger_cat                      | Probability: 0.002165
Class: Persian_cat                    | Probability: 0.000105
Class: Angora                         | Probability: 0.000074
Class: swab                           | Probability: 0.000071
Class: Madagascar_cat                 | Probability: 0.000064
Class: snow_leopard                   | Probability: 0.000040
Class: tile_roof                      | Probability: 0.000037
Class: indri                          | Probability: 0.000020
Class: leopard                        | Probability: 0.000016
Class: Siamese_cat                    | Probability: 0.000011
Class: ram                            | Probability: 0.000010
Class: crate                          | Probability: 0.000009

Model Inception-v3 - wyjaśnienieĀ¶

Chcemy wiedzieć dlaczego klasa Bernese_mountain_dog została uznana przez sieć neuronową za najbardziej prawdopodobną (to znaczy - ktĆ³re piksele obrazka o tym zadecydowały). W tym celu właśnie wykorzystamy LIME.

W jaki sposĆ³b działa LIME na obrazkach?

  1. Na wejściu wymagany jest oryginalny obrazek.
  2. Wejściowy obrazek jest delikatnie przekształcany wiele razy, dzięki czemu otrzymujemy wiele podobnych (ale nie takich samych!) obrazkĆ³w.
  3. Dodatkowo na wejście musimy podać funkcję, ktĆ³ra każdemu takiemu przekształceniu nada prawdopodobieństwo przynależności do danej klasy. Jest to wymagane ponieważ LIME jest niezależny od żadnych narzędzi i modeli.
InĀ [22]:
explainer = lime_image.LimeImageExplainer()
InĀ [23]:
explanation = explainer.explain_instance(
    image=lime_transformer(image_to_classify), 
    classifier_fn=partial(predict_batch, inception_v3),
    top_labels=5,
    num_samples=1000)

  0%|          | 0/1000 [00:00<?, ?it/s]

Mając te dane możemy teraz sprawdzić ktĆ³re kategorie są najbardziej prawdopodobne

InĀ [24]:
for index in explanation.top_labels:
    print(index_to_label[index])

Bernese_mountain_dog
EntleBucher
Appenzeller
Greater_Swiss_Mountain_dog
Gordon_setter

Zobaczmy co wpłynęło na wybranie Bernese_mountain_dog jako najbardziej prawdopodobnej klasy.

InĀ [25]:
image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[25]:
<matplotlib.image.AxesImage at 0x2428a6deed0>
No description has been provided for this image

Zadanie: zmień wartośćĀ NUM_FEATURES i zaobserwuj jak zmienia się mapowanieĀ¶

NUM_FEATURES najlepiej zmieniać w zakresie 1:50

InĀ [26]:
NUM_FEATURES = 10

image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=NUM_FEATURES,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[26]:
<matplotlib.image.AxesImage at 0x2428a769010>
No description has been provided for this image
InĀ [27]:
NUM_FEATURES = 30

image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=NUM_FEATURES,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[27]:
<matplotlib.image.AxesImage at 0x2428a7b8110>
No description has been provided for this image
InĀ [28]:
NUM_FEATURES = 1

image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=NUM_FEATURES,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[28]:
<matplotlib.image.AxesImage at 0x2428a6ca8a0>
No description has been provided for this image

The bigger NUM_FEATURES is the bigger area is marked. For too small value unimportant area is marked.

Zielone fragmenty oznaczają "superpiksele", ktĆ³re pozytywnie wpływają na predykowaną klasę. Czerwone fragmenty wpływają negatywnie.

Zadanie-pytanie: co to właściwie jest superpiksel?Ā¶

Superpixel is an area which is the most important part of this object to classify to it's class.

Zadanie-pytanie: czy jeden superpiksel ma odzwierciedlenie w jednym pikselu z obrazka?Ā¶

No, one superpixel is much bigger than just a pixel.

Zobaczmy jak to się prezentuje dla drugiej najbardziej prawdopodobnej klasy, czyli EntleBucher, ktĆ³ra jednak otrzymała jedyne 3.8%.

InĀ [29]:
image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[1],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[29]:
<matplotlib.image.AxesImage at 0x2428fd0e360>
No description has been provided for this image

Ustawiając wartości hide_rest oraz positive_only na True jesteśmy w stanie zostawić tylko te piksele, ktĆ³re potwierdzały przynależność do danej klasy
Musimy jednak pamiętać o przeskalowaniu rezultatu przy pomocy (boundaries).astype(np.uint8)

InĀ [30]:
image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[0],
    positive_only=True,
    negative_only=False,
    num_features=10,
    hide_rest=True)
boundaries = mark_boundaries(image, mask)
plt.imshow((boundaries).astype(np.uint8))
Out[30]:
<matplotlib.image.AxesImage at 0x2428fcf3740>
No description has been provided for this image

Możemy rĆ³wnież zostawić tylko te piksele, ktĆ³re zaprzeczały przynależności do danej klasy

InĀ [31]:
image, mask = explanation.get_image_and_mask(
    label=explanation.top_labels[0],
    positive_only=False,
    negative_only=True,
    num_features=10,
    hide_rest=True)
boundaries = mark_boundaries(image, mask)
cropped_image_ndarray = (boundaries).astype(np.uint8)
plt.imshow(cropped_image_ndarray)
Out[31]:
<matplotlib.image.AxesImage at 0x24290101c10>
No description has been provided for this image

A następnie sprawdzićĀ co model sądzi o tak wyciętym obrazku

InĀ [32]:
cropped_image_pil = Image.fromarray(cropped_image_ndarray)

get_prediction_probabilities(cropped_image_pil, inception_v3)

Class: lumbermill                     | Probability: 0.272693
Class: triceratops                    | Probability: 0.241274
Class: pedestal                       | Probability: 0.118357
Class: sundial                        | Probability: 0.081880
Class: fountain                       | Probability: 0.029933
Class: caldron                        | Probability: 0.029304
Class: African_grey                   | Probability: 0.018543
Class: vulture                        | Probability: 0.017247
Class: stone_wall                     | Probability: 0.013481
Class: cannon                         | Probability: 0.011382
Class: black_grouse                   | Probability: 0.009726
Class: frilled_lizard                 | Probability: 0.009235
Class: chain_saw                      | Probability: 0.007655
Class: grand_piano                    | Probability: 0.007258
Class: conch                          | Probability: 0.006042

I jak go teraz widzi model

InĀ [33]:
cropped_image_explanation = explainer.explain_instance(
    image=lime_transformer(cropped_image_pil), 
    classifier_fn=partial(predict_batch, inception_v3),
    top_labels=5,
    num_samples=1000)

  0%|          | 0/1000 [00:00<?, ?it/s]
InĀ [34]:
image, mask = cropped_image_explanation.get_image_and_mask(
    label=cropped_image_explanation.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[34]:
<matplotlib.image.AxesImage at 0x2428a6c9880>
No description has been provided for this image

Model Inception-v3 - porĆ³wnanie z AlexNetĀ¶

Przetestujmy działanie na innym modelu - AlexNet

InĀ [35]:
alexnet = models.alexnet(pretrained=True)

c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Downloading: "https://download.pytorch.org/models/alexnet-owt-7be5be79.pth" to C:\Users\Monika/.cache\torch\hub\checkpoints\alexnet-owt-7be5be79.pth
100%|ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ| 233M/233M [00:05<00:00, 47.6MB/s]
InĀ [36]:
explanation_alexnet = explainer.explain_instance(
    image=lime_transformer(image_to_classify), 
    classifier_fn=partial(predict_batch, alexnet),
    top_labels=5,
    num_samples=1000)

  0%|          | 0/1000 [00:00<?, ?it/s]
InĀ [37]:
for index_alex, index_inception in zip(explanation_alexnet.top_labels, explanation.top_labels):
    print(f"{index_to_label[index_alex]:30} | {index_to_label[index_inception]:30}")

Bernese_mountain_dog           | Bernese_mountain_dog          
EntleBucher                    | EntleBucher                   
Greater_Swiss_Mountain_dog     | Appenzeller                   
Appenzeller                    | Greater_Swiss_Mountain_dog    
basset                         | Gordon_setter

Jak widać, klasy nieco się rĆ³Å¼nią, ale TOP 1 pozostaje takie samo.

InĀ [38]:
image, mask = explanation_alexnet.get_image_and_mask(
    label=explanation_alexnet.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[38]:
<matplotlib.image.AxesImage at 0x242905e2030>
No description has been provided for this image

Wyjaśnienie dla AlexNet jak się można było spodziewać - teżĀ się rĆ³Å¼ni, jednak w dalszym ciągu do klasyfikacji psa istotny jest... pies :)

Zadanie: porĆ³wnaj predykcje obrazka dla modeli inception_v3 oraz alexnetĀ¶

InĀ [39]:
print("inception_v3")
get_prediction_probabilities(image_to_classify, inception_v3)
print()
print("alexnet")
get_prediction_probabilities(image_to_classify, alexnet)

inception_v3
Class: Bernese_mountain_dog           | Probability: 0.935930
Class: EntleBucher                    | Probability: 0.038448
Class: Appenzeller                    | Probability: 0.023756
Class: Greater_Swiss_Mountain_dog     | Probability: 0.001818
Class: Gordon_setter                  | Probability: 0.000009
Class: Blenheim_spaniel               | Probability: 0.000007
Class: English_springer               | Probability: 0.000002
Class: tabby                          | Probability: 0.000002
Class: robin                          | Probability: 0.000001
Class: guinea_pig                     | Probability: 0.000001
Class: amphibian                      | Probability: 0.000001
Class: Japanese_spaniel               | Probability: 0.000001
Class: African_grey                   | Probability: 0.000001
Class: Brittany_spaniel               | Probability: 0.000001
Class: toucan                         | Probability: 0.000001

alexnet
Class: Bernese_mountain_dog           | Probability: 0.400542
Class: EntleBucher                    | Probability: 0.208726
Class: Greater_Swiss_Mountain_dog     | Probability: 0.194558
Class: Appenzeller                    | Probability: 0.186891
Class: basset                         | Probability: 0.002400
Class: Cardigan                       | Probability: 0.002184
Class: Border_collie                  | Probability: 0.001509
Class: kelpie                         | Probability: 0.000398
Class: bluetick                       | Probability: 0.000314
Class: Blenheim_spaniel               | Probability: 0.000276
Class: collie                         | Probability: 0.000255
Class: borzoi                         | Probability: 0.000188
Class: Walker_hound                   | Probability: 0.000177
Class: Tibetan_mastiff                | Probability: 0.000171
Class: English_springer               | Probability: 0.000156

Zadanie domowe - wstępĀ¶

W folderze data znajduje sięĀ zdjęcie amfibii: title

InĀ [40]:
amphibious_vehicle = get_image("./data/amfibia.jpg")
InĀ [55]:
inception_v3 = models.inception_v3()

explanation_amhibious_vehicle_inception_v3 = explainer.explain_instance(
    image=lime_transformer(amphibious_vehicle), 
    classifier_fn=partial(predict_batch, inception_v3),
    top_labels=5,
    num_samples=1000)

image, mask = explanation_amhibious_vehicle_inception_v3.get_image_and_mask(
    label=explanation_amhibious_vehicle_inception_v3.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)

c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=Inception_V3_Weights.IMAGENET1K_V1`. You can also use `weights=Inception_V3_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)

  0%|          | 0/1000 [00:00<?, ?it/s]
Out[55]:
<matplotlib.image.AxesImage at 0x2428a299640>
No description has been provided for this image

Model inception_v3 jak i jego wyjaśnienie rzeczywiście sugerują amfibię jako najbardziej prawdopodobną klasę:

InĀ [56]:
for index in explanation_amhibious_vehicle_inception_v3.top_labels:
    print(index_to_label[index])

amphibian
convertible
racer
car_wheel
golfcart

Zadanie #1Ā¶

Użyj dwĆ³ch rĆ³Å¼nych sieci neuronowych (poza inception_v3, ktĆ³rego przykład jest powyżej) do wygenerowania wyjaśnień.
(skorzystaj z modułu torchvision: https://pytorch.org/vision/stable/models.html)

alexNetĀ¶

InĀ [53]:
alexnet = models.alexnet()

explanation_amhibious_vehicle_alexnet = explainer.explain_instance(
    image=lime_transformer(amphibious_vehicle), 
    classifier_fn=partial(predict_batch, alexnet),
    top_labels=5,
    num_samples=1000)

image, mask = explanation_amhibious_vehicle_alexnet.get_image_and_mask(
    label=explanation_amhibious_vehicle_alexnet.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)

  0%|          | 0/1000 [00:00<?, ?it/s]
Out[53]:
<matplotlib.image.AxesImage at 0x24296462ff0>
No description has been provided for this image
InĀ [54]:
for index in explanation_amhibious_vehicle_alexnet.top_labels:
    print(index_to_label[index])

ashcan
great_grey_owl
pickup
cardigan
weevil

GoogleNetĀ¶

InĀ [45]:
googlenet = models.googlenet(pretrained=True)

explanation_amhibious_vehicle_googlenet = explainer.explain_instance(
    image=lime_transformer(amphibious_vehicle), 
    classifier_fn=partial(predict_batch, googlenet),
    top_labels=5,
    num_samples=1000)

image, mask = explanation_amhibious_vehicle_googlenet.get_image_and_mask(
    label=explanation_amhibious_vehicle_googlenet.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)

c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=GoogLeNet_Weights.IMAGENET1K_V1`. You can also use `weights=GoogLeNet_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Downloading: "https://download.pytorch.org/models/googlenet-1378be20.pth" to C:\Users\Monika/.cache\torch\hub\checkpoints\googlenet-1378be20.pth
100%|ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ| 49.7M/49.7M [00:01<00:00, 50.1MB/s]

  0%|          | 0/1000 [00:00<?, ?it/s]
Out[45]:
<matplotlib.image.AxesImage at 0x2428a29bef0>
No description has been provided for this image
InĀ [46]:
for index in explanation_amhibious_vehicle_googlenet.top_labels:
    print(index_to_label[index])

pickup
tow_truck
jeep
amphibian
convertible

Zadanie #2Ā¶

Zmodyfikuj oryginalny obrazek w taki sposĆ³b, żeby najbardziej prawdopodobną klasą dla każdej z tych sieci nie była amfibia a jakiś inny pojazd (np. samochĆ³d). W tym celu możesz "zasłonić" czarnym kwadratem (wartość 0 w macierzy reprezentującej obraz) obszary istotne przy klasyfikacji.
Przydatną rzeczą będzie skorzystanie z opcji hide_rest w funkcji get_image_and_mask i pĆ³Åŗniejsza obrĆ³bka obrazu

InĀ [86]:
# Only the inception_v3 has the most probably amphibia
inception_v3 = models.inception_v3()

explanation_amhibious_vehicle_inception_v3 = explainer.explain_instance(
    image=lime_transformer(amphibious_vehicle), 
    classifier_fn=partial(predict_batch, inception_v3),
    top_labels=5,
    num_samples=1000)

c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\inception.py:43: FutureWarning: The default weight initialization of inception_v3 will be changed in future releases of torchvision. If you wish to keep the old behavior (which leads to long initialization times due to scipy/scipy#11299), please set init_weights=True.
  warnings.warn(

  0%|          | 0/1000 [00:00<?, ?it/s]
InĀ [109]:
image, mask = explanation_amhibious_vehicle_inception_v3.get_image_and_mask(
    label=explanation_amhibious_vehicle_inception_v3.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
Out[109]:
<matplotlib.image.AxesImage at 0x24298989e20>
No description has been provided for this image
InĀ [114]:
modified_image = image.copy()
positive_mask = mask.astype(bool)
modified_image[positive_mask] = [0, 0, 0]
plt.imshow(modified_image)
plt.axis('off')
plt.show()
No description has been provided for this image
InĀ [119]:
modified_image = Image.fromarray(modified_image)
InĀ [120]:
explanation_amhibious_vehicle_inception_v3_modified = explainer.explain_instance(
    image=lime_transformer(modified_image), 
    classifier_fn=partial(predict_batch, inception_v3),
    top_labels=5,
    num_samples=1000)

  0%|          | 0/1000 [00:00<?, ?it/s]
InĀ [121]:
for index in explanation_amhibious_vehicle_inception_v3_modified.top_labels:
    print(index_to_label[index])

cairn
bulbul
coral_fungus
tench
toilet_tissue

Zadanie #3Ā¶

Ponownie zmodyfikuj oryginalny obraz, ale tym razem zaszumiając go w losowy sposĆ³b (przykładowa implementacja: https://www.geeksforgeeks.org/add-a-salt-and-pepper-noise-to-an-image-with-python/). Czy najbardziej prawdopodobna klasa zmienia się wraz ze zmianą szumu? Przetestuj dla każdego z modeli.

InĀ [122]:
import random 
import cv2 

def add_noise(img): 
    row , col = img.shape 
    # white
    number_of_pixels = random.randint(300, 10000) 
    for i in range(number_of_pixels): 
        y_coord=random.randint(0, row - 1) 
        x_coord=random.randint(0, col - 1) 
        img[y_coord][x_coord] = 255
          
    # black
    number_of_pixels = random.randint(300 , 10000) 
    for i in range(number_of_pixels): 
        y_coord=random.randint(0, row - 1) 
        x_coord=random.randint(0, col - 1) 
        img[y_coord][x_coord] = 0
          
    return img 
  
img = cv2.imread('./data/amfibia.jpg', 
                 cv2.IMREAD_GRAYSCALE) 

cv2.imwrite('salt-and-pepper-amfibia.jpg', 
            add_noise(img))
Out[122]:
True
InĀ [123]:
amphibious_noise = get_image('salt-and-pepper-amfibia.jpg')
plt.imshow(amphibious_noise)
plt.axis('off')
plt.show()
No description has been provided for this image

Inception_v3Ā¶

InĀ [125]:
inception_v3 = models.inception_v3()

explanation_amhibious_vehicle_inception_v3 = explainer.explain_instance(
    image=lime_transformer(amphibious_noise), 
    classifier_fn=partial(predict_batch, inception_v3),
    top_labels=5,
    num_samples=1000)

  0%|          | 0/1000 [00:00<?, ?it/s]
InĀ [133]:
image, mask = explanation_amhibious_vehicle_inception_v3.get_image_and_mask(
    label=explanation_amhibious_vehicle_inception_v3.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
plt.axis('off')
plt.show()
No description has been provided for this image
InĀ [127]:
for index in explanation_amhibious_vehicle_inception_v3.top_labels:
    print(index_to_label[index])

Cardigan
bulbul
coral_fungus
tench
toilet_tissue

AlexNetĀ¶

InĀ [134]:
alexnet = models.alexnet()

explanation_amhibious_vehicle_alexnet = explainer.explain_instance(
    image=lime_transformer(amphibious_noise), 
    classifier_fn=partial(predict_batch, alexnet),
    top_labels=5,
    num_samples=1000)

image, mask = explanation_amhibious_vehicle_alexnet.get_image_and_mask(
    label=explanation_amhibious_vehicle_alexnet.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)
plt.axis('off')
plt.show()

  0%|          | 0/1000 [00:00<?, ?it/s]
No description has been provided for this image
InĀ [129]:
for index in explanation_amhibious_vehicle_alexnet.top_labels:
    print(index_to_label[index])

brambling
cricket
English_setter
triumphal_arch
head_cabbage

GoogLeNetĀ¶

InĀ [130]:
googlenet = models.googlenet(pretrained=True)

explanation_amhibious_vehicle_googlenet = explainer.explain_instance(
    image=lime_transformer(amphibious_noise), 
    classifier_fn=partial(predict_batch, googlenet),
    top_labels=5,
    num_samples=1000)

image, mask = explanation_amhibious_vehicle_googlenet.get_image_and_mask(
    label=explanation_amhibious_vehicle_googlenet.top_labels[0],
    positive_only=False,
    negative_only=False,
    num_features=10,
    hide_rest=False)
boundaries = mark_boundaries(image, mask)
plt.imshow(boundaries)

c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
c:\Users\Monika\AppData\Local\Programs\Python\Python312\Lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=GoogLeNet_Weights.IMAGENET1K_V1`. You can also use `weights=GoogLeNet_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)

  0%|          | 0/1000 [00:00<?, ?it/s]
Out[130]:
<matplotlib.image.AxesImage at 0x2429e77ee40>
No description has been provided for this image
InĀ [131]:
for index in explanation_amhibious_vehicle_googlenet.top_labels:
    print(index_to_label[index])

amphibian
jeep
tow_truck
snowplow
half_track

Using various models the most probably class is changing. Only GoogLeNet still is finding amphibian as first choice and futher other car types.